{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xfcSMZEIFA_l"
   },
   "source": [
    "# ML with TensorFlow Extended (TFX) -- Part 3\n",
    "The puprpose of this tutorial is to show how to do end-to-end ML with TFX libraries on Google Cloud Platform. This tutorial covers:\n",
    "1. Data analysis and schema generation with **TF Data Validation**.\n",
    "2. Data preprocessing with **TF Transform**.\n",
    "3. Model training with **TF Estimator**.\n",
    "4. Model evaluation with **TF Model Analysis**.\n",
    "\n",
    "This notebook has been tested in Jupyter on the Deep Learning VM.\n",
    "\n",
    "## Setup Cloud environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_data_validation as tfdv\n",
    "import tensorflow_transform as tft\n",
    "\n",
    "print('TF version: {}'.format(tf.__version__))\n",
    "print('TFT version: {}'.format(tft.__version__))\n",
    "print('TFDV version: {}'.format(tfdv.__version__))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT = 'cloud-training-demos'    # Replace with your PROJECT\n",
    "BUCKET = 'cloud-training-demos-ml'  # Replace with your BUCKET\n",
    "REGION = 'us-central1'              # Choose an available region for Cloud MLE\n",
    "\n",
    "import os\n",
    "\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['REGION'] = REGION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud config set project $PROJECT\n",
    "gcloud config set compute/region $REGION\n",
    "\n",
    "## ensure we predict locally with our current Python environment\n",
    "gcloud config set ml_engine/local_python `which python`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img valign=\"middle\" src=\"images/tfx.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l9u699PmHJXU"
   },
   "source": [
    "### UCI Adult Dataset: https://archive.ics.uci.edu/ml/datasets/adult\n",
    "Predict whether income exceeds $50K/yr based on census data. Also known as \"Census Income\" dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_DIR='gs://cloud-samples-data/ml-engine/census/data'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "ksuSTsysHfZV",
    "outputId": "87adfbf0-be77-4d81-9162-5a2f9feffd90"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "TRAIN_DATA_FILE = os.path.join(DATA_DIR, 'adult.data.csv')\n",
    "EVAL_DATA_FILE = os.path.join(DATA_DIR, 'adult.test.csv')\n",
    "!gcloud storage ls --long $TRAIN_DATA_FILE\n",    "!gcloud storage ls --long $EVAL_DATA_FILE"   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HEADER = ['age', 'workclass', 'fnlwgt', 'education', 'education_num',\n",
    "               'marital_status', 'occupation', 'relationship', 'race', 'gender',\n",
    "               'capital_gain', 'capital_loss', 'hours_per_week',\n",
    "               'native_country', 'income_bracket']\n",
    "\n",
    "TARGET_FEATURE_NAME = 'income_bracket'\n",
    "TARGET_LABELS = [' <=50K', ' >50K']\n",
    "WEIGHT_COLUMN_NAME = 'fnlwgt_scaled' # note that you changes the column name in tft\n",
    "\n",
    "RAW_SCHEMA_LOCATION = 'raw_schema.pbtxt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-qNjgoalG0Xu"
   },
   "source": [
    "## 3. Model Training\n",
    "For training the model, we use [TF Estimators](https://www.tensorflow.org/guide/estimators) APIs to train a premade DNNClassifier. We perform the following:\n",
    "1. Load the **transform schema**\n",
    "2. Use the transform schema to parse TFRecords in **input_fn**\n",
    "3. Use the transform schema to create **feature columns**\n",
    "4. Create a premade **DNNClassifier**\n",
    "5. **Train** the model\n",
    "6. Implement the **serving_input_fn** and apply the **transform logic**\n",
    "7. **Export** and test the saved model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U4eLEbV42XEz"
   },
   "source": [
    "### 3.1 Load transform output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREPROC_OUTPUT_DIR = 'gs://{}/census/tfx'.format(BUCKET)  # from 02_transform.ipynb\n",
    "TRANSFORM_ARTIFACTS_DIR = os.path.join(PREPROC_OUTPUT_DIR,'transform')\n",
    "TRANSFORMED_DATA_DIR = os.path.join(PREPROC_OUTPUT_DIR,'transformed')\n",
    "!gcloud storage ls $TRANSFORM_ARTIFACTS_DIR\n",    "!gcloud storage ls $TRANSFORMED_DATA_DIR"   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HR_d4-kBG0hI"
   },
   "outputs": [],
   "source": [
    "transform_output = tft.TFTransformOutput(TRANSFORM_ARTIFACTS_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bxt31_6M3dbl"
   },
   "source": [
    "### 3.2 TFRecords Input Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EQqf9dGF3dEe"
   },
   "outputs": [],
   "source": [
    "def make_input_fn(tfrecords_files, \n",
    "  batch_size, num_epochs=1, shuffle=False):\n",
    "\n",
    "  def input_fn():\n",
    "    dataset = tf.data.experimental.make_batched_features_dataset(\n",
    "      file_pattern=tfrecords_files,\n",
    "      batch_size=batch_size,\n",
    "      features=transform_output.transformed_feature_spec(),\n",
    "      label_key=TARGET_FEATURE_NAME,\n",
    "      reader=tf.data.TFRecordDataset,\n",
    "      num_epochs=num_epochs,\n",
    "      shuffle=shuffle\n",
    "    )\n",
    "    return dataset\n",
    "\n",
    "  return input_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "make_input_fn(TRANSFORMED_DATA_DIR+'/train*.tfrecords', 2, shuffle=False)()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vgQtTPiN4Td0"
   },
   "source": [
    "### 3.3 Create feature columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XzP3BUnU4bE5"
   },
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "def create_feature_columns():\n",
    "\n",
    "  feature_columns = []\n",
    "  transformed_features = transform_output.transformed_metadata.schema._schema_proto.feature\n",
    "\n",
    "  for feature in transformed_features:\n",
    "\n",
    "    if feature.name in [TARGET_FEATURE_NAME, WEIGHT_COLUMN_NAME]:\n",
    "      continue\n",
    "\n",
    "    if hasattr(feature, 'int_domain') and feature.int_domain.is_categorical:\n",
    "      vocab_size = feature.int_domain.max + 1\n",
    "      feature_columns.append(\n",
    "        tf.feature_column.embedding_column(\n",
    "          tf.feature_column.categorical_column_with_identity(\n",
    "            feature.name, num_buckets=vocab_size),\n",
    "            dimension = int(math.sqrt(vocab_size))))\n",
    "    else:\n",
    "      feature_columns.append(\n",
    "        tf.feature_column.numeric_column(feature.name))\n",
    "\n",
    "  return feature_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 275
    },
    "colab_type": "code",
    "id": "wAttQuXh-ZXR",
    "outputId": "4f998a45-7c41-460c-8d66-e59bf18a507b"
   },
   "outputs": [],
   "source": [
    "create_feature_columns()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Cr4iLJtK4bgv"
   },
   "source": [
    "### 3.4 Instantiate and Estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IKAGwUgP4Tkf"
   },
   "outputs": [],
   "source": [
    "def create_estimator(params, run_config):\n",
    "    \n",
    "  feature_columns = create_feature_columns()\n",
    "\n",
    "  estimator = tf.estimator.DNNClassifier(\n",
    "    weight_column=WEIGHT_COLUMN_NAME,\n",
    "    label_vocabulary=TARGET_LABELS,\n",
    "    feature_columns=feature_columns,\n",
    "    hidden_units=params.hidden_units,\n",
    "    config=run_config\n",
    "  )\n",
    "\n",
    "  return estimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iZIyscv84nXF"
   },
   "source": [
    "### 3.5 Implement train and evaluate experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "e3zLhoZj4nd4"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "def run_experiment(estimator, params, run_config, resume=False):\n",
    "  \n",
    "  tf.logging.set_verbosity(tf.logging.INFO)\n",
    "\n",
    "  if not resume: \n",
    "    if tf.gfile.Exists(run_config.model_dir):\n",
    "      print(\"Removing previous artifacts...\")\n",
    "      tf.gfile.DeleteRecursively(run_config.model_dir)\n",
    "  else:\n",
    "    print(\"Resuming training...\")\n",
    "\n",
    "  train_spec = tf.estimator.TrainSpec(\n",
    "      input_fn = make_input_fn(\n",
    "          TRANSFORMED_DATA_DIR+'/train*.tfrecords',\n",
    "          batch_size=params.batch_size,\n",
    "          num_epochs=None,\n",
    "          shuffle=True\n",
    "      ),\n",
    "      max_steps=params.max_steps\n",
    "  )\n",
    "\n",
    "  eval_spec = tf.estimator.EvalSpec(\n",
    "      input_fn = make_input_fn(\n",
    "          TRANSFORMED_DATA_DIR+'/eval*.tfrecords',\n",
    "          batch_size=params.batch_size,     \n",
    "      ),\n",
    "      start_delay_secs=0,\n",
    "      throttle_secs=0,\n",
    "      steps=None\n",
    "  )\n",
    "  \n",
    "  time_start = datetime.utcnow() \n",
    "  print(\"Experiment started at {}\".format(time_start.strftime(\"%H:%M:%S\")))\n",
    "  print(\".......................................\")\n",
    "  \n",
    "  tf.estimator.train_and_evaluate(\n",
    "    estimator=estimator,\n",
    "    train_spec=train_spec, \n",
    "    eval_spec=eval_spec)\n",
    "\n",
    "  time_end = datetime.utcnow() \n",
    "  print(\".......................................\")\n",
    "  print(\"Experiment finished at {}\".format(time_end.strftime(\"%H:%M:%S\")))\n",
    "  print(\"\")\n",
    "  \n",
    "  time_elapsed = time_end - time_start\n",
    "  print(\"Experiment elapsed time: {} seconds\".format(time_elapsed.total_seconds()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q1VBm6Bo4ntU"
   },
   "source": [
    "### 3.5 Run experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4mHEsbh94n0V"
   },
   "outputs": [],
   "source": [
    "MODELS_LOCATION = 'models/census'\n",
    "MODEL_NAME = 'dnn_classifier'\n",
    "model_dir = os.path.join(MODELS_LOCATION, MODEL_NAME)\n",
    "os.environ['MODEL_DIR'] = model_dir\n",
    "\n",
    "params = tf.contrib.training.HParams()\n",
    "params.hidden_units = [128, 64]\n",
    "params.dropout = 0.15\n",
    "params.batch_size =  128\n",
    "params.max_steps = 1000\n",
    "\n",
    "run_config = tf.estimator.RunConfig(\n",
    "    tf_random_seed=19831006,\n",
    "    save_checkpoints_steps=200, \n",
    "    keep_checkpoint_max=3, \n",
    "    model_dir=model_dir,\n",
    "    log_step_count_steps=10\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 4950
    },
    "colab_type": "code",
    "id": "9yMxrzCE5tWw",
    "outputId": "25c1dead-69c2-4b14-e987-bf21b056559b"
   },
   "outputs": [],
   "source": [
    "estimator = create_estimator(params, run_config)\n",
    "run_experiment(estimator, params, run_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pr-L_KM8LwmK"
   },
   "source": [
    "### 3.6 Export the model for serving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oBdB7mkvL43C"
   },
   "outputs": [],
   "source": [
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "\n",
    "def make_serving_input_receiver_fn():\n",
    "  from tensorflow_transform.tf_metadata import schema_utils\n",
    "\n",
    "  source_raw_schema = tfdv.load_schema_text(RAW_SCHEMA_LOCATION)\n",
    "  raw_feature_spec = schema_utils.schema_as_feature_spec(source_raw_schema).feature_spec\n",
    "  raw_feature_spec.pop(TARGET_FEATURE_NAME)\n",
    "  if WEIGHT_COLUMN_NAME in raw_feature_spec:\n",
    "    raw_feature_spec.pop(WEIGHT_COLUMN_NAME)\n",
    "\n",
    "\n",
    "  # Create the interface for the serving function with the raw features\n",
    "  raw_features = tf.estimator.export.build_parsing_serving_input_receiver_fn(raw_feature_spec)().features\n",
    "\n",
    "  receiver_tensors = {feature: tf.placeholder(shape=[None], dtype=raw_features[feature].dtype) \n",
    "    for feature in raw_features\n",
    "  }\n",
    "\n",
    "  receiver_tensors_expanded = {tensor: tf.reshape(receiver_tensors[tensor], (-1, 1)) \n",
    "    for tensor in receiver_tensors\n",
    "  }\n",
    "\n",
    "  # Apply the transform function \n",
    "  transformed_features = transform_output.transform_raw_features(receiver_tensors_expanded)\n",
    "\n",
    "  return tf.estimator.export.ServingInputReceiver(\n",
    "    transformed_features, receiver_tensors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 751
    },
    "colab_type": "code",
    "id": "_KcSJ6kEMG57",
    "outputId": "d0bcdb14-82ef-4b22-9484-90170f1b7984"
   },
   "outputs": [],
   "source": [
    "export_dir = os.path.join(model_dir, 'export')\n",
    "\n",
    "if tf.gfile.Exists(export_dir):\n",
    "    tf.gfile.DeleteRecursively(export_dir)\n",
    "        \n",
    "estimator.export_savedmodel(\n",
    "    export_dir_base=export_dir,\n",
    "    serving_input_receiver_fn=make_serving_input_receiver_fn\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1377
    },
    "colab_type": "code",
    "id": "XPFCTtSdQMd5",
    "outputId": "2ec32f67-47f5-4ef0-8a86-fe1797befdef"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "saved_models_base=${MODEL_DIR}/export/\n",
    "saved_model_dir=${MODEL_DIR}/export/$(ls ${saved_models_base} | tail -n 1)\n",
    "echo ${saved_model_dir}\n",
    "saved_model_cli show --dir=${saved_model_dir} --all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zFQcy8QDQ6Ad"
   },
   "source": [
    "### 3.7 Try out saved model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "colab_type": "code",
    "id": "0tacIgD5Q6HB",
    "outputId": "133588d7-1d0b-4a9f-fd88-b79d096f55b7"
   },
   "outputs": [],
   "source": [
    "export_dir = os.path.join(model_dir, 'export')\n",
    "tf.gfile.ListDirectory(export_dir)[-1]\n",
    "saved_model_dir = os.path.join(export_dir, tf.gfile.ListDirectory(export_dir)[-1])\n",
    "print(saved_model_dir)\n",
    "print()\n",
    "\n",
    "predictor_fn = tf.contrib.predictor.from_saved_model(\n",
    "    export_dir = saved_model_dir,\n",
    "    signature_def_key=\"predict\"\n",
    ")\n",
    "\n",
    "input = {\n",
    "        'age': [34.0],\n",
    "        'workclass': ['Private'],\n",
    "        'education': ['Doctorate'],\n",
    "        'education_num': [10.0],\n",
    "        'marital_status': ['Married-civ-spouse'],\n",
    "        'occupation': ['Prof-specialty'],\n",
    "        'relationship': ['Husband'],\n",
    "        'race': ['White'],\n",
    "        'gender': ['Male'],\n",
    "        'capital_gain': [0.0], \n",
    "        'capital_loss': [0.0], \n",
    "        'hours_per_week': [40.0],\n",
    "        'native_country':['Mexico']\n",
    "}\n",
    "\n",
    "print(input)\n",
    "print()\n",
    "output = predictor_fn(input)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.8 Deploy model to Cloud ML Engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%bash\n",
    "#MODEL_NAME=\"census\"\n",
    "#MODEL_VERSION=\"v1\"\n",
    "#MODEL_LOCATION=$(gcloud storage ls gs://${BUCKET}/census/dnn_classifier/export/exporter | tail -1)\n",
    "#gcloud ml-engine models create ${MODEL_NAME} --regions $REGION\n",
    "#gcloud ml-engine versions create ${MODEL_VERSION} --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --runtime-version 1.13"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.9 Export evaluation saved model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HEADER_DEFAULTS = [[0], [''], [0], [''], [0], [''], [''], [''], [''], [''],\n",
    "                   [0], [0], [0], [''], ['']]\n",
    "\n",
    "def make_eval_input_receiver_fn():\n",
    "  receiver_tensors = {'examples': tf.placeholder(dtype=tf.string, shape=[None])}\n",
    "  columns = tf.decode_csv(receiver_tensors['examples'], record_defaults=HEADER_DEFAULTS)\n",
    "  features = dict(zip(HEADER, columns))\n",
    "  print(features)\n",
    "\n",
    "  for feature_name in features:\n",
    "    if features[feature_name].dtype == tf.int32:\n",
    "      features[feature_name] = tf.cast(features[feature_name], tf.int64)\n",
    "    features[feature_name] = tf.reshape(features[feature_name], (-1, 1))\n",
    "\n",
    "  transformed_features = transform_output.transform_raw_features(features)\n",
    "  features.update(transformed_features)\n",
    "\n",
    "  return tfma.export.EvalInputReceiver(\n",
    "    features=features,\n",
    "    receiver_tensors=receiver_tensors,\n",
    "    labels=features[TARGET_FEATURE_NAME]\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow_model_analysis as tfma\n",
    "eval_model_dir = os.path.join(model_dir, \"export/evaluate\")\n",
    "if tf.gfile.Exists(eval_model_dir):\n",
    "    tf.gfile.DeleteRecursively(eval_model_dir)\n",
    "\n",
    "tfma.export.export_eval_savedmodel(\n",
    "        estimator=estimator,\n",
    "        export_dir_base=eval_model_dir,\n",
    "        eval_input_receiver_fn=make_eval_input_receiver_fn\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Gsi_Hsh89Cl7"
   },
   "source": [
    "## License"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0fOWx1yI9Dyn"
   },
   "source": [
    "Copyright 2019 Google LLC\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License.\n",
    "\n",
    "---\n",
    "This is not an official Google product. The sample code provided for educational purposes only.\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "02-tfx_end_to_end",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
